把这个选项进到主线了https://github.com/ggml-org/llama.cpp/commit/7f323a589f8684c0eb722e7309074cb5eac0c8b5

David's random thoughts

改一下llama.cpp把bs>=32时GPU通过PCIe读取主机内存里的模型的逻辑去掉，现在内存里的tensor全部由CPU处理，速度就正常多了。无论是用8G显存还是96G显存的pp性能都各自提升到10倍，远超纯CPU运行的性能。prefill现在可以跑出300 token/s的速度，基本达到日常可用水准。

把这个选项进到主线了

https://github.com/ggml-org/llama.cpp/commit/7f323a589f8684c0eb722e7309074cb5eac0c8b5

GitHub

Add `--no-op-offload` to improve `-ot` pp perf in MoE models like lla… · ggml-org/llama.cpp@7f323a5

…ma4 400B (#13386)

www.tg-me.com/us/David& 39;s random thoughts/com.david_random/579

1.9K viewsMay 12 at 10:09

tg-me.com/david_random/579

Create: 2025-05-12
Last Update: 2025-06-27 10:18:38

把这个选项进到主线了

https://github.com/ggml-org/llama.cpp/commit/7f323a589f8684c0eb722e7309074cb5eac0c8b5

BY David's random thoughts

Share with your friend now:
tg-me.com/david_random/579

David& 39;s random thoughts Telegram | DID YOU KNOW?

Dump Scam in Leaked Telegram Chat

A leaked Telegram discussion by 50 so-called crypto influencers has exposed the extraordinary steps they take in order to profit on the back off unsuspecting defi investors. According to a leaked screenshot of the chat, an elaborate plan to defraud defi investors using the worthless “$Few” tokens had been hatched. $Few tokens would be airdropped to some of the influencers who in turn promoted these to unsuspecting followers on Twitter.

The S&P 500 slumped 1.8% on Monday and Tuesday, thanks to China Evergrande, the Chinese property company that looks like it is ready to default on its more-than $300 billion in debt. Cries of the next Lehman Brothers—or maybe the next Silverado?—echoed through the canyons of Wall Street as investors prepared for the worst.